ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenes

The two popular datasets ScanRefer [16] and ReferIt3D [3] connect natural language to real-world 3D data. In this paper, we curate a large-scale and complementary dataset extending both the aforementioned ones by associating all objects mentioned in a referential sentence to their underlying instances inside a 3D scene. Specifically, our Scan Entities in 3D (ScanEnts3D) dataset provides explicit correspondences between 369k objects across 84k natural referential sentences, covering 705 real-world scenes. Crucially, we show that by incorporating intuitive losses that enable learning from this novel dataset, we can significantly improve the performance of several recently introduced neural listening architectures, including improving the SoTA in both the Nr3D and ScanRefer benchmarks by 4.3% and 5.0%, respectively. Moreover, we experiment with competitive baselines and recent methods for the task of language generation and show that, as with neural listeners, 3D neural speakers can also noticeably benefit by training with ScanEnts3D, including improving the SoTA by 13.2 CIDEr points on the Nr3D benchmark. Overall, our carefully conducted experimental studies strongly support the conclusion that, by learning on ScanEnts3D, commonly used visio-linguistic 3D architectures can become more efficient and interpretable in their generalization without needing to provide these newly collected annotations at test time. The project's webpage is https://scanents3d.github.io/ .Comment: The project's webpage is https://scanents3d.github.io

Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation

Diffusion probabilistic models (DPMs) have become a popular approach to conditional generation, due to their promising results and support for cross-modal synthesis. A key desideratum in conditional synthesis is to achieve high correspondence between the conditioning input and generated output. Most existing methods learn such relationships implicitly, by incorporating the prior into the variational lower bound. In this work, we take a different route -- we explicitly enhance input-output connections by maximizing their mutual information. To this end, we introduce a Conditional Discrete Contrastive Diffusion (CDCD) loss and design two contrastive diffusion mechanisms to effectively incorporate it into the denoising process, combining the diffusion training and contrastive learning for the first time by connecting it with the conventional variational objectives. We demonstrate the efficacy of our approach in evaluations with diverse multimodal conditional synthesis tasks: dance-to-music generation, text-to-image synthesis, as well as class-conditioned image synthesis. On each, we enhance the input-output correspondence and achieve higher or competitive general synthesis quality. Furthermore, the proposed approach improves the convergence of diffusion models, reducing the number of required diffusion steps by more than 35% on two benchmarks, significantly increasing the inference speed.Comment: ICLR 2023. Project at https://github.com/L-YeZhu/CDC

Intuitive, Interactive Beard and Hair Synthesis with Generative Models

We present an interactive approach to synthesizing realistic variations in facial hair in images, ranging from subtle edits to existing hair to the addition of complex and challenging hair in images of clean-shaven subjects. To circumvent the tedious and computationally expensive tasks of modeling, rendering and compositing the 3D geometry of the target hairstyle using the traditional graphics pipeline, we employ a neural network pipeline that synthesizes realistic and detailed images of facial hair directly in the target image in under one second. The synthesis is controlled by simple and sparse guide strokes from the user defining the general structural and color properties of the target hairstyle. We qualitatively and quantitatively evaluate our chosen method compared to several alternative approaches. We show compelling interactive editing results with a prototype user interface that allows novice users to progressively refine the generated image to match their desired hairstyle, and demonstrate that our approach also allows for flexible and high-fidelity scalp hair synthesis.Comment: To be presented in the 2020 Conference on Computer Vision and Pattern Recognition (CVPR 2020, Oral Presentation). Supplementary video can be seen at: https://www.youtube.com/watch?v=v4qOtBATrv

Quantized GAN for Complex Music Generation from Dance Videos

We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal framework that generates complex musical samples conditioned on dance videos. Our proposed framework takes dance video frames and human body motion as input, and learns to generate music samples that plausibly accompany the corresponding input. Unlike most existing conditional music generation works that generate specific types of mono-instrumental sounds using symbolic audio representations (e.g., MIDI), and that heavily rely on pre-defined musical synthesizers, in this work we generate dance music in complex styles (e.g., pop, breakdancing, etc.) by employing a Vector Quantized (VQ) audio representation, and leverage both its generality and the high abstraction capacity of its symbolic and continuous counterparts. By performing an extensive set of experiments on multiple datasets, and following a comprehensive evaluation protocol, we assess the generative quality of our approach against several alternatives. The quantitative results, which measure the music consistency, beats correspondence, and music diversity, clearly demonstrate the effectiveness of our proposed method. Last but not least, we curate a challenging dance-music dataset of in-the-wild TikTok videos, which we use to further demonstrate the efficacy of our approach in real-world applications - and which we hope to serve as a starting point for relevant future research

Navigating Scholarly Development Through a CoP: #generational-lenses

CoPs provide opportunity for professional identity exploration. Extant research neglects how multiple identities, including generational memberships, influence development. This study explores professional identity development of scholars within a multigenerational CoP

The α1D-adrenergic receptor is expressed intracellularly and coupled to increases in intracellular calcium and reactive oxygen species in human aortic smooth muscle cells

Background: The cellular localization of the α1D-adrenergic receptor (α1D-AR) is controversial. Studies in heterologous cell systems have shown that this receptor is expressed in intracellular compartments. Other studies show that dimerization with other ARs promotes the cell surface expression of the α1D-AR. To assess the cellular localization in vascular smooth muscle cells, we developed an adenoviral vector for the efficient expression of a GFP labeled α1D-AR. We also measured cellular localization with immunocytochemistry. Intracellular calcium levels, measurement of reactive oxygen species and contraction of the rat aorta were used as measures of functional activity. Results: The adenovirally expressed α1D-AR was expressed in intracellular compartments in human aortic smooth muscle cells. The intracellular localization of the α1D-AR was also demonstrated with immunocytochemistry using an α1D-AR specific antibody. RT-PCR analysis detected mRNA transcripts corresponding to the α1A-α1B- and α1D-ARs in these aortic smooth muscle cells. Therefore, the presence of the other α1-ARs, and the potential for dimerization with these receptors, does not alter the intracellular expression of the α1D-AR. Despite the predominant intracellular localization in vascular smooth muscle cells, the α1D-AR remained signaling competent and mediated the phenylephrine-induced increases in intracellular calcium. The α1D-AR also was coupled to the generation of reactive oxygen species in smooth muscle cells. There is evidence from heterologous systems that the α1D-AR heterodimerizes with the β2-AR and that desensitization of the β2-AR results in α1D-AR desensitization. In the rat aorta, desensitization of the β2-AR had no effect on contractile responses mediated by the α1D-AR. Conclusion: Our results suggest that the dimerization of the α1D-AR with other ARs does not alter the cellular expression or functional response characteristics of the α1D-AR

The α\u3csub\u3e1D\u3c/sub\u3e-Adrenergic Receptor Is Expressed Intracellularly and Coupled to Increases in Intracellular Calcium and Reactive Oxygen Species in Human Aortic Smooth Muscle Cells

Background: The cellular localization of the α1D-adrenergic receptor (α1D-AR) is controversial. Studies in heterologous cell systems have shown that this receptor is expressed in intracellular compartments. Other studies show that dimerization with other ARs promotes the cell surface expression of the α1D-AR. To assess the cellular localization in vascular smooth muscle cells, we developed an adenoviral vector for the efficient expression of a GFP labeled α1D-AR. We also measured cellular localization with immunocytochemistry. Intracellular calcium levels, measurement of reactive oxygen species and contraction of the rat aorta were used as measures of functional activity. Results: The adenovirally expressed α1D-AR was expressed in intracellular compartments in human aortic smooth muscle cells. The intracellular localization of the α1D-AR was also demonstrated with immunocytochemistry using an α1D-AR specific antibody. RT-PCR analysis detected mRNA transcripts corresponding to the α1A-α1B- and α1D-ARs in these aortic smooth muscle cells. Therefore, the presence of the other α1-ARs, and the potential for dimerization with these receptors, does not alter the intracellular expression of the α1D-AR. Despite the predominant intracellular localization in vascular smooth muscle cells, the α1D-AR remained signaling competent and mediated the phenylephrine-induced increases in intracellular calcium. The α1D-AR also was coupled to the generation of reactive oxygen species in smooth muscle cells. There is evidence from heterologous systems that the α1D-AR heterodimerizes with the β2-AR and that desensitization of the β2-AR results in α1D-AR desensitization. In the rat aorta, desensitization of the β2-AR had no effect on contractile responses mediated by the α1D-AR. Conclusion: Our results suggest that the dimerization of the α1D-AR with other ARs does not alter the cellular expression or functional response characteristics of the α1D-AR

R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis

Recent research explosion on Neural Radiance Field (NeRF) shows the encouraging potential to represent complex scenes with neural networks. One major drawback of NeRF is its prohibitive inference time: Rendering a single pixel requires querying the NeRF network hundreds of times. To resolve it, existing efforts mainly attempt to reduce the number of required sampled points. However, the problem of iterative sampling still exists. On the other hand, Neural Light Field (NeLF) presents a more straightforward representation over NeRF in novel view synthesis -- the rendering of a pixel amounts to one single forward pass without ray-marching. In this work, we present a deep residual MLP network (88 layers) to effectively learn the light field. We show the key to successfully learning such a deep NeLF network is to have sufficient data, for which we transfer the knowledge from a pre-trained NeRF model via data distillation. Extensive experiments on both synthetic and real-world scenes show the merits of our method over other counterpart algorithms. On the synthetic scenes, we achieve 26-35x FLOPs reduction (per camera ray) and 28-31x runtime speedup, meanwhile delivering significantly better (1.4-2.8 dB average PSNR improvement) rendering quality than NeRF without any customized implementation tricks.Comment: Project: https://snap-research.github.io/R2

Unexpected Learning: Development of the CoP and Its Members #generational-shift

Our research explores how multigenerational CoPs may provide graduate students, particularly doctoral students, the space to explore and develop their professional identities and find their scholarly voices